{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "在自然语言处理任务中，词向量（word2vec）是表示自然语言里单词的一种方法，即把每个词都表示为一个N维空间内的点，即一个高维空间内的向量。通过这种方法，实现把自然语言计算转换为向量计算。\n",
    "\n",
    "如 **图1** 所示的词向量计算任务中，先把每个词（如queen，king等）转换成一个高维空间的向量，这些向量在一定意义上可以代表这个词的语义信息。再通过计算这些向量之间的距离，就可以计算出词语之间的关联关系，从而达到让计算机像计算数值一样去计算自然语言的目的。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/8b2a4e3cf6b146a8890b11e8531b04f8ee36e94624b44109bcfc7b064030a107\" width=\"800\" ></center>\n",
    "<center>图1：词向量计算示意图</center>\n",
    "<br></br>\n",
    "\n",
    "因此，大部分词向量模型都需要回答两个问题：\n",
    "\n",
    "1. **如何把词转换为向量?**\n",
    "2. **如何让向量具有语义信息?**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 如何把词转换为向量\n",
    "\n",
    "自然语言单词是离散信号，比如“我”、“ 爱”、“人工智能”。如何把每个离散的单词转换为一个向量？通常情况下，我们可以维护一个如 **图2** 所示的查询表。表中每一行都存储了一个特定词语的向量值，每一列的第一个元素都代表着这个词本身，以便于我们进行词和向量的映射（如“我”对应的向量值为 [0.3，0.5，0.7，0.9，-0.2，0.03] ）。给定任何一个或者一组单词，我们都可以通过查询这个excel，实现把单词转换为向量的目的，这个查询和替换过程称之为Embedding Lookup。\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/36c579bebf8a441caee97ac40a8b18928d7010bec07f4924ac62a7c9328d0331\" width=\"700\" ></center>\n",
    "<center><br>图2：词向量查询表</br></center>\n",
    "<br></br>\n",
    "\n",
    "上述过程也可以使用一个字典数据结构实现。事实上如果不考虑计算效率，使用字典实现上述功能是个不错的选择。然而在进行神经网络计算的过程中，需要大量的算力，常常要借助特定硬件（如GPU）满足训练速度的需求。但是GPU仅支持以张量（tensor）为单位计算，因此在实际场景中，我们需要把Embedding Lookup的过程转换为张量计算，如 **图3** 所示。\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/a3368ba3d61c49d88ee7257c9a02ebf81d0e5e9a84654719a0dfb7302b545053\" width=\"600\" ></center>\n",
    "<center><br>图3：张量计算示意图</br></center>\n",
    "<br></br>\n",
    "\n",
    "假设对于句子\"我，爱，人工，智能\"，把Embedding Lookup的过程转换为张量计算的流程如下：\n",
    "\n",
    "1. 通过查询字典，先把句子中的单词转换成一个ID（通常是一个大于等于0的整数），这个单词到ID的映射关系可以根据需求自定义（如**图3**中，我=>1, 人工=>2，智能=>3，...）。\n",
    "\n",
    "2. 得到ID后，再把每个ID转换成一个固定长度的向量。假设字典的词表中有50000个词，那么，对于单词“我”，就可以用一个50000维的向量来表示。由于“我”的ID是1，因此这个向量的第一个元素是1，其他元素都是0（[1，0，0，…，0]）；同样对于单词“人工”，第二个元素是1，其他元素都是0。用这种方式就实现了用一个张量表示一个单词。由于每个单词的向量表示都只有一个元素为1，而其他元素为0，因此我们称上述过程为One-Hot Encoding。\n",
    "\n",
    "3. 经过One-Hot Encoding后，句子“我，爱，人工，智能”就被转换成为了一个形状为 4 * 50000 的张量，记为$V$。在这个张量里共有4行、50000列，从上到下，每一行分别代表了“我”、“爱”、“人工”、“智能”四个单词的One-Hot Encoding。最后，我们把这个张量$V$和另外一个稠密张量$W$相乘，其中$W$张量的形状为50000 * 128（50000表示词表大小，128表示每个词的向量大小）。经过张量乘法，我们就得到了一个4*128的张量，从而完成了把单词表示成向量的目的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 如何让向量具有语义信息\n",
    "\n",
    "得到每个单词的向量表示后，我们需要思考下一个问题：比如在多数情况下，“香蕉”和“橘子”更加相似，而“香蕉”和“句子”就没有那么相似；同时，“香蕉”和“食物”、“水果”的相似程度可能介于“橘子”和“句子”之间。那么我们该如何让存储的词向量具备这样的语义信息呢？\n",
    "\n",
    "我们先学习自然语言处理领域的一个小技巧。在自然语言处理研究中，科研人员通常有一个共识：使用一个单词的上下文来了解这个单词的语义，比如：\n",
    "\n",
    " >“苹果手机质量不错，就是价格有点贵。”\n",
    " >\n",
    " >“这个苹果很好吃，非常脆。”\n",
    " >\n",
    " >“菠萝质量也还行，但是不如苹果支持的APP多。”\n",
    ">\n",
    "在上面的句子中，我们通过上下文可以推断出第一个“苹果”指的是苹果手机，第二个“苹果”指的是水果苹果，而第三个“菠萝”指的应该也是一个手机。事实上，在自然语言处理领域，使用上下文描述一个词语或者元素的语义是一个常见且有效的做法。我们可以使用同样的方式训练词向量，让这些词向量具备表示语义信息的能力。\n",
    "\n",
    "2013年，Mikolov提出的经典word2vec算法就是通过上下文来学习语义信息。word2vec包含两个经典模型，CBOW（Continuous Bag-of-Words）和Skip-gram，如 **图4** 所示。\n",
    "\n",
    "- **CBOW**：通过上下文的词向量预测中心词。\n",
    "- **Skip-gram**：根据中心词预测上下文。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/1d9c790c000a4fa88812d75346672c2efed16dbb7a0c4e0ca3e42421f89b7724\" width=\"800\" ></center>\n",
    "<center><br>图4：CBOW和Skip-gram语义学习示意图</br></center>\n",
    "<br></br>\n",
    "\n",
    "假设有一个句子“Pineapples are spiked and yellow”，两个模型的预测方式如下：\n",
    "\n",
    "- 在**CBOW**中，先在句子中选定一个中心词，并把其他词作为这个中心词的上下文。如 **图4** CBOW所示，把“spiked”作为中心词，把“Pineapples are and yellow”作为中心词的上下文。在学习过程中，使用上下文的词向量预测中心词，这样中心词的语义就被传递到上下文的词向量中，如“spiked”=>“pineapple”，从而达到学习语义信息的目的。\n",
    "\n",
    "- 在**Skip-gram**中，同样先选定一个中心词，并把其他词作为这个中心词的上下文。如 **图4** Skip-gram所示，把“spiked”作为中心词，把“Pineapples are and yellow”作为中心词的上下文。不同的是，在学习过程中，使用中心词的词向量去预测上下文，从而达到学习语义信息的目的。\n",
    "\n",
    "------\n",
    "**说明：**\n",
    "\n",
    "一般来说，CBOW比Skip-gram训练速度快，训练过程更加稳定，原因是CBOW使用上下文average的方式进行训练，每个训练step会见到更多样本。而在生僻字（出现频率低的字）处理上，skip-gram比CBOW效果更好，原因是skip-gram不会刻意回避生僻字。\n",
    "\n",
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## CBOW和Skip-gram的算法实现\n",
    "\n",
    "CBOW和Skip-gram的算法实现如 **图5** 所示，下面我们以Skip-gram为例，详细介绍算法实现过程。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/eee9dc52cd4f4be5b74c568df2e302859be16460fca44960aa2d788ea8b9328c\" width=\"700\" ></center>\n",
    "<center><br>图5：CBOW和Skip-gram的算法实现</br></center>\n",
    "<br></br>\n",
    "\n",
    "如 **图5** 所示，Skip-gram是一个具有3层结构的神经网络，分别是：\n",
    "\n",
    "- **Input Layer（输入层）**：接收一个one-hot张量 $V \\in R^{1 \\times \\text{vocab\\_size}}$ 作为网络的输入，里面存储着当前句子中心词的one-hot表示。\n",
    "- **Hidden Layer（隐藏层）**：将张量$V$乘以一个word embedding张量$W^1 \\in R^{\\text{vocab\\_size} \\times \\text{embed\\_size}}$，并把结果作为隐藏层的输出，得到一个形状为$R^{1 \\times \\text{embed\\_size}}$的张量，里面存储着当前句子中心词的词向量。\n",
    "- **Output Layer（输出层）**：将隐藏层的结果乘以另一个word embedding张量$W^2 \\in R^{\\text{embed\\_size} \\times \\text{vocab\\_size}}$，得到一个形状为$R^{1 \\times \\text{vocab\\_size}}$的张量。这个张量经过softmax变换后，就得到了使用当前中心词对上下文的预测结果。根据这个softmax的结果，我们就可以去训练词向量模型。\n",
    "\n",
    "在实际操作中，使用一个滑动窗口（一般情况下，长度是奇数），从左到右开始扫描当前句子。每个扫描出来的片段被当成一个小句子，每个小句子中间的词被认为是中心词，其余的词被认为是这个中心词的上下文。\n",
    "\n",
    "### Skip-gram的理想实现\n",
    "\n",
    "使用神经网络实现Skip-gram中，模型接收的输入应该有2个不同的tensor：\n",
    "\n",
    "- 代表中心词的tensor：假设我们称之为center_words $V$，一般来说，这个tensor是一个形状为[batch_size, vocab_size]的one-hot tensor，表示在一个mini-batch中，每个中心词的ID。\n",
    "\n",
    "- 代表目标词的tensor：假设我们称之为target_words $T$，一般来说，这个tensor是一个形状为[batch_size, 1]的整型tensor，这个tensor中的每个元素是一个[0, vocab_size-1]的值，代表目标词的ID。\n",
    "\n",
    "在理想情况下，我们可以使用一个简单的方式实现skip-gram。即把需要预测的每个目标词都当成一个标签，把skip-gram当成一个大规模分类任务进行网络构建，过程如下：\n",
    "\n",
    "1. 声明一个形状为[vocab_size, embedding_size]的张量，作为需要学习的词向量，记为$W_0$。对于给定的输入$V$，使用向量乘法，将$V$乘以$W_0$，这样就得到了一个形状为[batch_size, embedding_size]的张量，记为$H=V*W_0$。这个张量$H$就可以看成是经过词向量查表后的结果。\n",
    "1. 声明另外一个需要学习的参数$W_1$，这个参数的形状为[embedding_size, vocab_size]。将上一步得到的$H$去乘以$W_1$，得到一个新的tensor $O=H*W_1$，此时的$O$是一个形状为[batch_size, vocab_size]的tensor，表示当前这个mini-batch中的每个中心词预测出的目标词的概率。\n",
    "1. 使用softmax函数对mini-batch中每个中心词的预测结果做归一化，即可完成网络构建。\n",
    "\n",
    "### Skip-gram的实际实现\n",
    "\n",
    "然而在实际情况中，vocab_size通常很大（几十万甚至几百万），导致$W_0$和$W_1$也会非常大。对于$W_0$而言，所参与的矩阵运算并不是通过一个矩阵乘法实现，而是通过指定ID，对参数$W_0$进行访存的方式获取。然而对$W_1$而言，仍要处理一个非常大的矩阵运算（计算过程非常缓慢，需要消耗大量的内存/显存）。为了缓解这个问题，通常采取负采样（negative_sampling）的方式来近似模拟多分类任务。\n",
    "\n",
    "假设有一个中心词$c$和一个目标词正样本$t_p$。在Skip-gram的理想实现里，需要最大化使用$c$预测$t_p$的概率。在使用softmax学习时，需要最大化$t_p$的预测概率，同时最小化其他词表中词的预测概率。之所以计算缓慢，是因为需要对词表中的所有词都计算一遍。然而我们还可以使用另一种方法，就是随机从词表中选择几个代表词，通过最小化这几个代表词的概率，去近似最小化整体的预测概率。比如，先指定一个中心词（如“人工”）和一个目标词正样本（如“智能”），再随机在词表中采样几个目标词负样本（如“日本”，“喝茶”等）。有了这些内容，我们的skip-gram模型就变成了一个二分类任务。对于目标词正样本，我们需要最大化它的预测概率；对于目标词负样本，我们需要最小化它的预测概率。通过这种方式，我们就可以完成计算加速。上述做法，我们称之为负采样。\n",
    "\n",
    "在实现的过程中，通常会让模型接收3个tensor输入：\n",
    "\n",
    "- 代表中心词的tensor：假设我们称之为center_words $V$，一般来说，这个tensor是一个形状为[batch_size, vocab_size]的one-hot tensor，表示在一个mini-batch中每个中心词具体的ID。\n",
    "\n",
    "- 代表目标词的tensor：假设我们称之为target_words $T$，一般来说，这个tensor同样是一个形状为[batch_size, vocab_size]的one-hot tensor，表示在一个mini-batch中每个目标词具体的ID。\n",
    "\n",
    "- 代表目标词标签的tensor：假设我们称之为labels $L$，一般来说，这个tensor是一个形状为[batch_size, 1]的tensor，每个元素不是0就是1（0：负样本，1：正样本）。\n",
    "\n",
    "模型训练过程如下：\n",
    "1. 用$V$去查询$W_0$，用$T$去查询$W_1$，分别得到两个形状为[batch_size, embedding_size]的tensor，记为$H_1$和$H_2$。\n",
    "1. 点乘这两个tensor，最终得到一个形状为[batch_size]的tensor  $O = [O_i = \\sum_j H_0[i,j] * H_1[i,j]]_{i=1}^{batch\\_size}$。\n",
    "1. 使用sigmoid函数作用在$O$上，将上述点乘的结果归一化为一个0-1的概率值，作为预测概率，根据标签信息label训练这个模型即可。\n",
    "\n",
    "在结束模型训练之后，一般使用$W_0$作为最终要使用的词向量，可以用$W_0$提供的向量表示。通过向量点乘的方式，计算两个不同词之间的相似度。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "##  使用飞桨实现Skip-gram\n",
    "\n",
    "接下来我们将学习使用飞桨实现Skio-gram模型的方法。在飞桨中，不同深度学习模型的训练过程基本一致，流程如下：\n",
    "\n",
    "1. **数据处理**：选择需要使用的数据，并做好必要的预处理工作。\n",
    "\n",
    "2. **网络定义**：使用飞桨定义好网络结构，包括输入层，中间层，输出层，损失函数和优化算法。\n",
    "\n",
    "3. **网络训练**：将准备好的数据送入神经网络进行学习，并观察学习的过程是否正常，如损失函数值是否在降低，也可以打印一些中间步骤的结果出来等。\n",
    "\n",
    "4. **网络评估**：使用测试集合测试训练好的神经网络，看看训练效果如何。\n",
    "\n",
    "在数据处理前，需要先加载飞桨平台（如果用户在本地使用，请确保已经安装飞桨）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import io\n",
    "import os\n",
    "import sys\n",
    "import requests\n",
    "from collections import OrderedDict \n",
    "import math\n",
    "import random\n",
    "import numpy as np\n",
    "import paddle\n",
    "import paddle.fluid as fluid\n",
    "\n",
    "from paddle.fluid.dygraph.nn import Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 数据处理\n",
    "\n",
    "首先，找到一个合适的语料用于训练word2vec模型。我们选择text8数据集，这个数据集里包含了大量从维基百科收集到的英文语料，我们可以通过如下代码下载数据集，下载后的文件被保存在当前目录的text8.txt文件内。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#下载语料用来训练word2vec\n",
    "def download():\n",
    "    #可以从百度云服务器下载一些开源数据集（dataset.bj.bcebos.com）\n",
    "    corpus_url = \"https://dataset.bj.bcebos.com/word2vec/text8.txt\"\n",
    "    #使用python的requests包下载数据集到本地\n",
    "    web_request = requests.get(corpus_url)\n",
    "    corpus = web_request.content\n",
    "    #把下载后的文件存储在当前目录的text8.txt文件内\n",
    "    with open(\"./text8.txt\", \"wb\") as f:\n",
    "        f.write(corpus)\n",
    "    f.close()\n",
    "\n",
    "download()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "接下来，把下载的语料读取到程序里，并打印前500个字符看看语料的样子，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " anarchism originated as a term of abuse first used against early working class radicals including the diggers of the english revolution and the sans culottes of the french revolution whilst the term is still used in a pejorative way to describe any act that used violent means to destroy the organization of society it has also been taken up as a positive label by self defined anarchists the word anarchism is derived from the greek without archons ruler chief king anarchism as a political philoso\n"
     ]
    }
   ],
   "source": [
    "#读取text8数据\n",
    "def load_text8():\n",
    "    with open(\"./text8.txt\", \"r\") as f:\n",
    "        corpus = f.read().strip(\"\\n\")\n",
    "    f.close()\n",
    "\n",
    "    return corpus\n",
    "\n",
    "corpus = load_text8()\n",
    "\n",
    "#打印前500个字符，简要看一下这个语料的样子\n",
    "print(corpus[:500])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "一般来说，在自然语言处理中，需要先对语料进行切词。对于英文来说，可以比较简单地直接使用空格进行切词，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['anarchism', 'originated', 'as', 'a', 'term', 'of', 'abuse', 'first', 'used', 'against', 'early', 'working', 'class', 'radicals', 'including', 'the', 'diggers', 'of', 'the', 'english', 'revolution', 'and', 'the', 'sans', 'culottes', 'of', 'the', 'french', 'revolution', 'whilst', 'the', 'term', 'is', 'still', 'used', 'in', 'a', 'pejorative', 'way', 'to', 'describe', 'any', 'act', 'that', 'used', 'violent', 'means', 'to', 'destroy', 'the']\n"
     ]
    }
   ],
   "source": [
    "#对语料进行预处理（分词）\n",
    "def data_preprocess(corpus):\n",
    "    #由于英文单词出现在句首的时候经常要大写，所以我们把所有英文字符都转换为小写，\n",
    "    #以便对语料进行归一化处理（Apple vs apple等）\n",
    "    corpus = corpus.strip().lower()\n",
    "    corpus = corpus.split(\" \")\n",
    "\n",
    "    return corpus\n",
    "\n",
    "corpus = data_preprocess(corpus)\n",
    "print(corpus[:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "在经过切词后，需要对语料进行统计，为每个词构造ID。一般来说，可以根据每个词在语料中出现的频次构造ID，频次越高，ID越小，便于对词典进行管理。代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#构造词典，统计每个词的频率，并根据频率将每个词转换为一个整数id\n",
    "def build_dict(corpus):\n",
    "    #首先统计每个不同词的频率（出现的次数），使用一个词典记录\n",
    "    word_freq_dict = dict()\n",
    "    for word in corpus:\n",
    "        if word not in word_freq_dict:\n",
    "            word_freq_dict[word] = 0\n",
    "        word_freq_dict[word] += 1\n",
    "\n",
    "    #将这个词典中的词，按照出现次数排序，出现次数越高，排序越靠前\n",
    "    #一般来说，出现频率高的高频词往往是：I，the，you这种代词，而出现频率低的词，往往是一些名词，如：nlp\n",
    "    word_freq_dict = sorted(word_freq_dict.items(), key = lambda x:x[1], reverse = True)\n",
    "    \n",
    "    #构造3个不同的词典，分别存储，\n",
    "    #每个词到id的映射关系：word2id_dict\n",
    "    #每个id出现的频率：word2id_freq\n",
    "    #每个id到词典映射关系：id2word_dict\n",
    "    word2id_dict = dict()\n",
    "    word2id_freq = dict()\n",
    "    id2word_dict = dict()\n",
    "\n",
    "    #按照频率，从高到低，开始遍历每个单词，并为这个单词构造一个独一无二的id\n",
    "    for word, freq in word_freq_dict:\n",
    "        curr_id = len(word2id_dict)\n",
    "        word2id_dict[word] = curr_id\n",
    "        word2id_freq[word2id_dict[word]] = freq\n",
    "        id2word_dict[curr_id] = word\n",
    "\n",
    "    return word2id_freq, word2id_dict, id2word_dict\n",
    "\n",
    "word2id_freq, word2id_dict, id2word_dict = build_dict(corpus)\n",
    "vocab_size = len(word2id_freq)\n",
    "print(\"there are totoally %d different words in the corpus\" % vocab_size)\n",
    "for _, (word, word_id) in zip(range(50), word2id_dict.items()):\n",
    "    print(\"word %s, its id %d, its word freq %d\" % (word, word_id, word2id_freq[word_id]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "得到word2id词典后，我们还需要进一步处理原始语料，把每个词替换成对应的ID，便于神经网络进行处理，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "17005207 tokens in the corpus\n",
      "[5233, 3080, 11, 5, 194, 1, 3133, 45, 58, 155, 127, 741, 476, 10571, 133, 0, 27349, 1, 0, 102, 854, 2, 0, 15067, 58112, 1, 0, 150, 854, 3580, 0, 194, 10, 190, 58, 4, 5, 10712, 214, 6, 1324, 104, 454, 19, 58, 2731, 362, 6, 3672, 0]\n"
     ]
    }
   ],
   "source": [
    "#把语料转换为id序列\n",
    "def convert_corpus_to_id(corpus, word2id_dict):\n",
    "    #使用一个循环，将语料中的每个词替换成对应的id，以便于神经网络进行处理\n",
    "    corpus = [word2id_dict[word] for word in corpus]\n",
    "    return corpus\n",
    "\n",
    "corpus = convert_corpus_to_id(corpus, word2id_dict)\n",
    "print(\"%d tokens in the corpus\" % len(corpus))\n",
    "print(corpus[:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "接下来，需要使用二次采样法处理原始文本。二次采样法的主要思想是降低高频词在语料中出现的频次，从而优化整个词表的词向量训练效果，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8744272 tokens in the corpus\n",
      "[5233, 3080, 3133, 58, 155, 741, 10571, 133, 27349, 102, 854, 2, 15067, 58112, 150, 854, 3580, 194, 10, 10712, 214, 1324, 58, 2731, 3672, 708, 371, 26, 40, 539, 97, 1423, 2757, 567, 686, 7088, 5233, 1052, 44611, 2877, 792, 186, 5233, 200, 602, 1134, 2621, 8983, 279, 4147]\n"
     ]
    }
   ],
   "source": [
    "#使用二次采样算法（subsampling）处理语料，强化训练效果\n",
    "def subsampling(corpus, word2id_freq):\n",
    "    \n",
    "    #这个discard函数决定了一个词会不会被替换，这个函数是具有随机性的，每次调用结果不同\n",
    "    #如果一个词的频率很大，那么它被遗弃的概率就很大\n",
    "    def discard(word_id):\n",
    "        return random.uniform(0, 1) < 1 - math.sqrt(\n",
    "            1e-4 / word2id_freq[word_id] * len(corpus))\n",
    "\n",
    "    corpus = [word for word in corpus if not discard(word)]\n",
    "    return corpus\n",
    "\n",
    "corpus = subsampling(corpus, word2id_freq)\n",
    "print(\"%d tokens in the corpus\" % len(corpus))\n",
    "print(corpus[:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "在完成语料数据预处理之后，需要构造训练数据。根据上面的描述，我们需要使用一个滑动窗口对语料从左到右扫描，在每个窗口内，中心词需要预测它的上下文，并形成训练数据。\n",
    "\n",
    "在实际操作中，由于词表往往很大（50000，100000等），对大词表的一些矩阵运算（如softmax）需要消耗巨大的资源，因此可以通过负采样的方式模拟softmax的结果。具体来说，给定一个中心词和一个需要预测的上下文词，把这个上下文词作为正样本；通过词表随机采样的方式，选择若干个负样本。这样就把一个大规模分类问题转化为一个2分类问题，通过这种方式优化计算速度，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "center_word anarchism, target originated, label 1\n",
      "center_word anarchism, target viteri, label 0\n",
      "center_word anarchism, target deedles, label 0\n",
      "center_word anarchism, target anticapitalism, label 0\n",
      "center_word anarchism, target prohvet, label 0\n",
      "center_word anarchism, target abuse, label 1\n",
      "center_word anarchism, target liszt, label 0\n",
      "center_word anarchism, target troostite, label 0\n",
      "center_word anarchism, target schoonheyt, label 0\n",
      "center_word anarchism, target sephiroticum, label 0\n",
      "center_word originated, target anarchism, label 1\n",
      "center_word originated, target petyo, label 0\n",
      "center_word originated, target qbert, label 0\n",
      "center_word originated, target nanyuan, label 0\n",
      "center_word originated, target mailers, label 0\n",
      "center_word originated, target abuse, label 1\n",
      "center_word originated, target hauptring, label 0\n",
      "center_word originated, target nja, label 0\n",
      "center_word originated, target taiho, label 0\n",
      "center_word originated, target northrop, label 0\n",
      "center_word abuse, target anarchism, label 1\n",
      "center_word abuse, target madaba, label 0\n",
      "center_word abuse, target arness, label 0\n",
      "center_word abuse, target brightest, label 0\n",
      "center_word abuse, target bikur, label 0\n",
      "center_word abuse, target originated, label 1\n",
      "center_word abuse, target venio, label 0\n",
      "center_word abuse, target preiddeu, label 0\n",
      "center_word abuse, target uomini, label 0\n",
      "center_word abuse, target hammerschmidt, label 0\n",
      "center_word abuse, target used, label 1\n",
      "center_word abuse, target sarnen, label 0\n",
      "center_word abuse, target gichi, label 0\n",
      "center_word abuse, target rfe, label 0\n",
      "center_word abuse, target bodhr, label 0\n",
      "center_word abuse, target against, label 1\n",
      "center_word abuse, target grossi, label 0\n",
      "center_word abuse, target cauer, label 0\n",
      "center_word abuse, target kuomintang, label 0\n",
      "center_word abuse, target guarneri, label 0\n",
      "center_word used, target abuse, label 1\n",
      "center_word used, target interlinear, label 0\n",
      "center_word used, target dawlati, label 0\n",
      "center_word used, target semester, label 0\n",
      "center_word used, target gullwing, label 0\n",
      "center_word used, target against, label 1\n",
      "center_word used, target candidly, label 0\n",
      "center_word used, target unyes, label 0\n",
      "center_word used, target mediagenic, label 0\n",
      "center_word used, target millionaire, label 0\n"
     ]
    }
   ],
   "source": [
    "#构造数据，准备模型训练\n",
    "#max_window_size代表了最大的window_size的大小，程序会根据max_window_size从左到右扫描整个语料\n",
    "#negative_sample_num代表了对于每个正样本，我们需要随机采样多少负样本用于训练，\n",
    "#一般来说，negative_sample_num的值越大，训练效果越稳定，但是训练速度越慢。 \n",
    "def build_data(corpus, word2id_dict, word2id_freq, max_window_size = 3, negative_sample_num = 4):\n",
    "    \n",
    "    #使用一个list存储处理好的数据\n",
    "    dataset = []\n",
    "\n",
    "    #从左到右，开始枚举每个中心点的位置\n",
    "    for center_word_idx in range(len(corpus)):\n",
    "        #以max_window_size为上限，随机采样一个window_size，这样会使得训练更加稳定\n",
    "        window_size = random.randint(1, max_window_size)\n",
    "        #当前的中心词就是center_word_idx所指向的词\n",
    "        center_word = corpus[center_word_idx]\n",
    "\n",
    "        #以当前中心词为中心，左右两侧在window_size内的词都可以看成是正样本\n",
    "        positive_word_range = (max(0, center_word_idx - window_size), min(len(corpus) - 1, center_word_idx + window_size))\n",
    "        positive_word_candidates = [corpus[idx] for idx in range(positive_word_range[0], positive_word_range[1]+1) if idx != center_word_idx]\n",
    "\n",
    "        #对于每个正样本来说，随机采样negative_sample_num个负样本，用于训练\n",
    "        for positive_word in positive_word_candidates:\n",
    "            #首先把（中心词，正样本，label=1）的三元组数据放入dataset中，\n",
    "            #这里label=1表示这个样本是个正样本\n",
    "            dataset.append((center_word, positive_word, 1))\n",
    "\n",
    "            #开始负采样\n",
    "            i = 0\n",
    "            while i < negative_sample_num:\n",
    "                negative_word_candidate = random.randint(0, vocab_size-1)\n",
    "\n",
    "                if negative_word_candidate not in positive_word_candidates:\n",
    "                    #把（中心词，正样本，label=0）的三元组数据放入dataset中，\n",
    "                    #这里label=0表示这个样本是个负样本\n",
    "                    dataset.append((center_word, negative_word_candidate, 0))\n",
    "                    i += 1\n",
    "    \n",
    "    return dataset\n",
    "\n",
    "dataset = build_data(corpus, word2id_dict, word2id_freq)\n",
    "for _, (center_word, target_word, label) in zip(range(50), dataset):\n",
    "    print(\"center_word %s, target %s, label %d\" % (id2word_dict[center_word],\n",
    "                                                   id2word_dict[target_word], label))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    " 训练数据准备好后，把训练数据都组装成mini-batch，并准备输入到网络中进行训练，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       0., 0., 0., 0., 0., 1., 0., 0., 0.], dtype=float32))\r"
     ]
    }
   ],
   "source": [
    "#构造mini-batch，准备对模型进行训练\n",
    "#我们将不同类型的数据放到不同的tensor里，便于神经网络进行处理\n",
    "#并通过numpy的array函数，构造出不同的tensor来，并把这些tensor送入神经网络中进行训练\n",
    "def build_batch(dataset, batch_size, epoch_num):\n",
    "    \n",
    "    #center_word_batch缓存batch_size个中心词\n",
    "    center_word_batch = []\n",
    "    #target_word_batch缓存batch_size个目标词（可以是正样本或者负样本）\n",
    "    target_word_batch = []\n",
    "    #label_batch缓存了batch_size个0或1的标签，用于模型训练\n",
    "    label_batch = []\n",
    "\n",
    "    for epoch in range(epoch_num):\n",
    "        #每次开启一个新epoch之前，都对数据进行一次随机打乱，提高训练效果\n",
    "        random.shuffle(dataset)\n",
    "        \n",
    "        for center_word, target_word, label in dataset:\n",
    "            #遍历dataset中的每个样本，并将这些数据送到不同的tensor里\n",
    "            center_word_batch.append([center_word])\n",
    "            target_word_batch.append([target_word])\n",
    "            label_batch.append(label)\n",
    "\n",
    "            #当样本积攒到一个batch_size后，我们把数据都返回回来\n",
    "            #在这里我们使用numpy的array函数把list封装成tensor\n",
    "            #并使用python的迭代器机制，将数据yield出来\n",
    "            #使用迭代器的好处是可以节省内存\n",
    "            if len(center_word_batch) == batch_size:\n",
    "                yield np.array(center_word_batch).astype(\"int64\"), \\\n",
    "                    np.array(target_word_batch).astype(\"int64\"), \\\n",
    "                    np.array(label_batch).astype(\"float32\")\n",
    "                center_word_batch = []\n",
    "                target_word_batch = []\n",
    "                label_batch = []\n",
    "\n",
    "    if len(center_word_batch) > 0:\n",
    "        yield np.array(center_word_batch).astype(\"int64\"), \\\n",
    "            np.array(target_word_batch).astype(\"int64\"), \\\n",
    "            np.array(label_batch).astype(\"float32\")\n",
    "\n",
    "for _, batch in zip(range(10), build_batch(dataset, 128, 3)):\n",
    "    print(batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 网络定义\n",
    "\n",
    "定义skip-gram的网络结构，用于模型训练。在飞桨动态图中，对于任意网络，都需要定义一个继承自fluid.dygraph.Layer的类来搭建网络结构、参数等数据的声明。同时需要在forward函数中定义网络的计算逻辑。值得注意的是，我们仅需要定义网络的前向计算逻辑，飞桨会自动完成神经网络的反向计算，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#定义skip-gram训练网络结构\n",
    "#这里我们使用的是paddlepaddle的1.6.1版本\n",
    "#一般来说，在使用fluid训练的时候，我们需要通过一个类来定义网络结构，这个类继承了fluid.dygraph.Layer\n",
    "class SkipGram(fluid.dygraph.Layer):\n",
    "    def __init__(self, name_scope, vocab_size, embedding_size, init_scale=0.1):\n",
    "        #name_scope定义了这个类某个具体实例的名字，以便于区分不同的实例（模型）\n",
    "        #vocab_size定义了这个skipgram这个模型的词表大小\n",
    "        #embedding_size定义了词向量的维度是多少\n",
    "        #init_scale定义了词向量初始化的范围，一般来说，比较小的初始化范围有助于模型训练\n",
    "        super(SkipGram, self).__init__(name_scope)\n",
    "        self.vocab_size = vocab_size\n",
    "        self.embedding_size = embedding_size\n",
    "\n",
    "        #使用paddle.fluid.dygraph提供的Embedding函数，构造一个词向量参数\n",
    "        #这个参数的大小为：[self.vocab_size, self.embedding_size]\n",
    "        #数据类型为：float32\n",
    "        #这个参数的名称为：embedding_para\n",
    "        #这个参数的初始化方式为在[-init_scale, init_scale]区间进行均匀采样\n",
    "        self.embedding = Embedding(\n",
    "            self.full_name(),\n",
    "            size=[self.vocab_size, self.embedding_size],\n",
    "            dtype='float32',\n",
    "            param_attr=fluid.ParamAttr(\n",
    "                name='embedding_para',\n",
    "                initializer=fluid.initializer.UniformInitializer(\n",
    "                    low=-0.5/embedding_size, high=0.5/embedding_size)))\n",
    "\n",
    "        #使用paddle.fluid.dygraph提供的Embedding函数，构造另外一个词向量参数\n",
    "        #这个参数的大小为：[self.vocab_size, self.embedding_size]\n",
    "        #数据类型为：float32\n",
    "        #这个参数的名称为：embedding_para\n",
    "        #这个参数的初始化方式为在[-init_scale, init_scale]区间进行均匀采样\n",
    "        #跟上面不同的是，这个参数的名称跟上面不同，因此，\n",
    "        #embedding_out_para和embedding_para虽然有相同的shape，但是权重不共享\n",
    "        self.embedding_out = Embedding(\n",
    "            self.full_name(),\n",
    "            size=[self.vocab_size, self.embedding_size],\n",
    "            dtype='float32',\n",
    "            param_attr=fluid.ParamAttr(\n",
    "                name='embedding_out_para',\n",
    "                initializer=fluid.initializer.UniformInitializer(\n",
    "                    low=-0.5/embedding_size, high=0.5/embedding_size)))\n",
    "\n",
    "    #定义网络的前向计算逻辑\n",
    "    #center_words是一个tensor（mini-batch），表示中心词\n",
    "    #target_words是一个tensor（mini-batch），表示目标词\n",
    "    #label是一个tensor（mini-batch），表示这个词是正样本还是负样本（用0或1表示）\n",
    "    #用于在训练中计算这个tensor中对应词的同义词，用于观察模型的训练效果\n",
    "    def forward(self, center_words, target_words, label):\n",
    "        #首先，通过embedding_para（self.embedding）参数，将mini-batch中的词转换为词向量\n",
    "        #这里center_words和eval_words_emb查询的是一个相同的参数\n",
    "        #而target_words_emb查询的是另一个参数\n",
    "        center_words_emb = self.embedding(center_words)\n",
    "        target_words_emb = self.embedding_out(target_words)\n",
    "\n",
    "        #center_words_emb = [batch_size, embedding_size]\n",
    "        #target_words_emb = [batch_size, embedding_size]\n",
    "        #我们通过点乘的方式计算中心词到目标词的输出概率，并通过sigmoid函数估计这个词是正样本还是负样本的概率。\n",
    "        word_sim = fluid.layers.elementwise_mul(center_words_emb, target_words_emb)\n",
    "        word_sim = fluid.layers.reduce_sum(word_sim, dim = -1)\n",
    "        pred = fluid.layers.sigmoid(word_sim)\n",
    "\n",
    "        #通过估计的输出概率定义损失函数\n",
    "        loss = fluid.layers.sigmoid_cross_entropy_with_logits(word_sim, label)\n",
    "        loss = fluid.layers.reduce_mean(loss)\n",
    "\n",
    "        #返回前向计算的结果，飞桨会通过backward函数自动计算出反向结果。\n",
    "        return pred, loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 网络训练\n",
    "\n",
    "完成网络定义后，就可以启动模型训练。我们定义每隔100步打印一次loss，以确保当前的网络是正常收敛的。同时，我们每隔1000步观察一下skip-gram计算出来的同义词（使用 embedding的乘积），可视化网络训练效果，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 100, loss 0.693\n",
      "step 200, loss 0.693\n",
      "step 300, loss 0.693\n",
      "step 400, loss 0.693\n",
      "step 500, loss 0.691\n",
      "step 600, loss 0.687\n",
      "step 700, loss 0.680\n",
      "step 800, loss 0.671\n",
      "step 900, loss 0.663\n",
      "step 1000, loss 0.644\n",
      "step 1100, loss 0.633\n",
      "step 1200, loss 0.603\n",
      "step 1300, loss 0.589\n",
      "step 1400, loss 0.567\n",
      "step 1500, loss 0.528\n",
      "step 1600, loss 0.502\n",
      "step 1700, loss 0.519\n",
      "step 1800, loss 0.498\n",
      "step 1900, loss 0.483\n",
      "step 2000, loss 0.462\n",
      "step 2100, loss 0.460\n",
      "step 2200, loss 0.428\n",
      "step 2300, loss 0.392\n",
      "step 2400, loss 0.408\n",
      "step 2500, loss 0.400\n",
      "step 2600, loss 0.391\n",
      "step 2700, loss 0.408\n",
      "step 2800, loss 0.379\n",
      "step 2900, loss 0.340\n",
      "step 3000, loss 0.299\n",
      "step 3100, loss 0.332\n",
      "step 3200, loss 0.349\n",
      "step 3300, loss 0.333\n",
      "step 3400, loss 0.302\n",
      "step 3500, loss 0.321\n",
      "step 3600, loss 0.290\n",
      "step 3700, loss 0.302\n",
      "step 3800, loss 0.293\n",
      "step 3900, loss 0.262\n",
      "step 4000, loss 0.256\n",
      "step 4100, loss 0.281\n",
      "step 4200, loss 0.249\n",
      "step 4300, loss 0.305\n",
      "step 4400, loss 0.259\n",
      "step 4500, loss 0.275\n",
      "step 4600, loss 0.233\n",
      "step 4700, loss 0.279\n",
      "step 4800, loss 0.289\n",
      "step 4900, loss 0.223\n",
      "step 5000, loss 0.241\n",
      "step 5100, loss 0.228\n",
      "step 5200, loss 0.231\n",
      "step 5300, loss 0.278\n",
      "step 5400, loss 0.280\n",
      "step 5500, loss 0.230\n",
      "step 5600, loss 0.182\n",
      "step 5700, loss 0.252\n",
      "step 5800, loss 0.237\n",
      "step 5900, loss 0.187\n",
      "step 6000, loss 0.267\n",
      "step 6100, loss 0.208\n",
      "step 6200, loss 0.224\n",
      "step 6300, loss 0.165\n",
      "step 6400, loss 0.239\n",
      "step 6500, loss 0.231\n",
      "step 6600, loss 0.233\n",
      "step 6700, loss 0.254\n",
      "step 6800, loss 0.293\n",
      "step 6900, loss 0.185\n",
      "step 7000, loss 0.235\n",
      "step 7100, loss 0.211\n",
      "step 7200, loss 0.238\n",
      "step 7300, loss 0.223\n",
      "step 7400, loss 0.205\n",
      "step 7500, loss 0.295\n",
      "step 7600, loss 0.241\n",
      "step 7700, loss 0.260\n",
      "step 7800, loss 0.212\n",
      "step 7900, loss 0.220\n",
      "step 8000, loss 0.236\n",
      "step 8100, loss 0.177\n",
      "step 8200, loss 0.213\n",
      "step 8300, loss 0.252\n",
      "step 8400, loss 0.193\n",
      "step 8500, loss 0.203\n",
      "step 8600, loss 0.235\n",
      "step 8700, loss 0.238\n",
      "step 8800, loss 0.292\n",
      "step 8900, loss 0.264\n",
      "step 9000, loss 0.216\n",
      "step 9100, loss 0.225\n",
      "step 9200, loss 0.247\n",
      "step 9300, loss 0.229\n",
      "step 9400, loss 0.195\n",
      "step 9500, loss 0.177\n",
      "step 9600, loss 0.241\n",
      "step 9700, loss 0.224\n",
      "step 9800, loss 0.193\n",
      "step 9900, loss 0.253\n",
      "step 10000, loss 0.173\n",
      "for word one, the similar word is one\n",
      "for word one, the similar word is five\n",
      "for word one, the similar word is two\n",
      "for word one, the similar word is nine\n",
      "for word one, the similar word is minutes\n",
      "for word she, the similar word is she\n",
      "for word she, the similar word is growth\n",
      "for word she, the similar word is travel\n",
      "for word she, the similar word is gameplay\n",
      "for word she, the similar word is arrived\n",
      "for word chip, the similar word is chip\n",
      "for word chip, the similar word is sadayoshi\n",
      "for word chip, the similar word is polices\n",
      "for word chip, the similar word is comforted\n",
      "for word chip, the similar word is benefitting\n",
      "step 10100, loss 0.242\n",
      "step 10200, loss 0.188\n",
      "step 10300, loss 0.212\n",
      "step 10400, loss 0.257\n",
      "step 10500, loss 0.224\n",
      "step 10600, loss 0.205\n",
      "step 10700, loss 0.230\n",
      "step 10800, loss 0.241\n",
      "step 10900, loss 0.223\n",
      "step 11000, loss 0.254\n",
      "step 11100, loss 0.192\n",
      "step 11200, loss 0.178\n",
      "step 11300, loss 0.252\n",
      "step 11400, loss 0.229\n",
      "step 11500, loss 0.246\n",
      "step 11600, loss 0.248\n",
      "step 11700, loss 0.213\n",
      "step 11800, loss 0.264\n",
      "step 11900, loss 0.248\n",
      "step 12000, loss 0.207\n",
      "step 12100, loss 0.211\n",
      "step 12200, loss 0.270\n",
      "step 12300, loss 0.257\n",
      "step 12400, loss 0.215\n",
      "step 12500, loss 0.225\n",
      "step 12600, loss 0.201\n",
      "step 12700, loss 0.250\n",
      "step 12800, loss 0.216\n",
      "step 12900, loss 0.211\n",
      "step 13000, loss 0.284\n",
      "step 13100, loss 0.213\n",
      "step 13200, loss 0.235\n",
      "step 13300, loss 0.208\n",
      "step 13400, loss 0.234\n",
      "step 13500, loss 0.269\n",
      "step 13600, loss 0.220\n",
      "step 13700, loss 0.248\n",
      "step 13800, loss 0.226\n",
      "step 13900, loss 0.260\n",
      "step 14000, loss 0.251\n",
      "step 14100, loss 0.242\n",
      "step 14200, loss 0.210\n",
      "step 14300, loss 0.219\n",
      "step 14400, loss 0.212\n",
      "step 14500, loss 0.232\n",
      "step 14600, loss 0.228\n",
      "step 14700, loss 0.220\n",
      "step 14800, loss 0.186\n",
      "step 14900, loss 0.284\n",
      "step 15000, loss 0.221\n",
      "step 15100, loss 0.224\n",
      "step 15200, loss 0.196\n",
      "step 15300, loss 0.157\n",
      "step 15400, loss 0.242\n",
      "step 15500, loss 0.225\n",
      "step 15600, loss 0.251\n",
      "step 15700, loss 0.256\n",
      "step 15800, loss 0.221\n",
      "step 15900, loss 0.237\n",
      "step 16000, loss 0.236\n",
      "step 16100, loss 0.227\n",
      "step 16200, loss 0.189\n",
      "step 16300, loss 0.190\n",
      "step 16400, loss 0.245\n",
      "step 16500, loss 0.174\n",
      "step 16600, loss 0.241\n",
      "step 16700, loss 0.260\n",
      "step 16800, loss 0.171\n",
      "step 16900, loss 0.172\n",
      "step 17000, loss 0.238\n",
      "step 17100, loss 0.216\n",
      "step 17200, loss 0.186\n",
      "step 17300, loss 0.216\n",
      "step 17400, loss 0.167\n",
      "step 17500, loss 0.236\n",
      "step 17600, loss 0.268\n",
      "step 17700, loss 0.231\n",
      "step 17800, loss 0.326\n",
      "step 17900, loss 0.243\n",
      "step 18000, loss 0.200\n",
      "step 18100, loss 0.228\n",
      "step 18200, loss 0.197\n",
      "step 18300, loss 0.190\n",
      "step 18400, loss 0.285\n",
      "step 18500, loss 0.186\n",
      "step 18600, loss 0.226\n",
      "step 18700, loss 0.251\n",
      "step 18800, loss 0.233\n",
      "step 18900, loss 0.203\n",
      "step 19000, loss 0.268\n",
      "step 19100, loss 0.249\n",
      "step 19200, loss 0.187\n",
      "step 19300, loss 0.207\n",
      "step 19400, loss 0.210\n",
      "step 19500, loss 0.184\n",
      "step 19600, loss 0.265\n",
      "step 19700, loss 0.155\n",
      "step 19800, loss 0.204\n",
      "step 19900, loss 0.263\n",
      "step 20000, loss 0.266\n",
      "for word one, the similar word is one\n",
      "for word one, the similar word is seven\n",
      "for word one, the similar word is nine\n",
      "for word one, the similar word is six\n",
      "for word one, the similar word is two\n",
      "for word she, the similar word is she\n",
      "for word she, the similar word is separatismo\n",
      "for word she, the similar word is akad\n",
      "for word she, the similar word is limassol\n",
      "for word she, the similar word is heightened\n",
      "for word chip, the similar word is chip\n",
      "for word chip, the similar word is aababb\n",
      "for word chip, the similar word is broadnax\n",
      "for word chip, the similar word is todi\n",
      "for word chip, the similar word is okoboji\n",
      "step 20100, loss 0.189\n",
      "step 20200, loss 0.209\n",
      "step 20300, loss 0.263\n",
      "step 20400, loss 0.252\n",
      "step 20500, loss 0.264\n",
      "step 20600, loss 0.257\n",
      "step 20700, loss 0.249\n",
      "step 20800, loss 0.320\n",
      "step 20900, loss 0.226\n",
      "step 21000, loss 0.207\n",
      "step 21100, loss 0.333\n",
      "step 21200, loss 0.271\n",
      "step 21300, loss 0.194\n",
      "step 21400, loss 0.224\n",
      "step 21500, loss 0.196\n",
      "step 21600, loss 0.233\n",
      "step 21700, loss 0.191\n",
      "step 21800, loss 0.251\n",
      "step 21900, loss 0.181\n",
      "step 22000, loss 0.245\n",
      "step 22100, loss 0.192\n",
      "step 22200, loss 0.199\n",
      "step 22300, loss 0.213\n",
      "step 22400, loss 0.263\n",
      "step 22500, loss 0.184\n",
      "step 22600, loss 0.290\n",
      "step 22700, loss 0.214\n",
      "step 22800, loss 0.209\n",
      "step 22900, loss 0.219\n",
      "step 23000, loss 0.233\n",
      "step 23100, loss 0.170\n",
      "step 23200, loss 0.189\n",
      "step 23300, loss 0.200\n",
      "step 23400, loss 0.211\n",
      "step 23500, loss 0.221\n",
      "step 23600, loss 0.289\n",
      "step 23700, loss 0.349\n",
      "step 23800, loss 0.251\n",
      "step 23900, loss 0.236\n",
      "step 24000, loss 0.274\n",
      "step 24100, loss 0.226\n",
      "step 24200, loss 0.177\n",
      "step 24300, loss 0.286\n",
      "step 24400, loss 0.237\n",
      "step 24500, loss 0.250\n",
      "step 24600, loss 0.185\n",
      "step 24700, loss 0.210\n",
      "step 24800, loss 0.222\n",
      "step 24900, loss 0.243\n",
      "step 25000, loss 0.288\n",
      "step 25100, loss 0.285\n",
      "step 25200, loss 0.244\n",
      "step 25300, loss 0.178\n",
      "step 25400, loss 0.212\n",
      "step 25500, loss 0.157\n",
      "step 25600, loss 0.290\n",
      "step 25700, loss 0.308\n",
      "step 25800, loss 0.221\n",
      "step 25900, loss 0.254\n",
      "step 26000, loss 0.212\n",
      "step 26100, loss 0.178\n",
      "step 26200, loss 0.296\n",
      "step 26300, loss 0.245\n",
      "step 26400, loss 0.173\n",
      "step 26500, loss 0.137\n",
      "step 26600, loss 0.209\n",
      "step 26700, loss 0.249\n",
      "step 26800, loss 0.237\n",
      "step 26900, loss 0.207\n",
      "step 27000, loss 0.197\n",
      "step 27100, loss 0.213\n",
      "step 27200, loss 0.226\n"
     ]
    }
   ],
   "source": [
    "#开始训练，定义一些训练过程中需要使用的超参数\n",
    "batch_size = 512\n",
    "epoch_num = 3\n",
    "embedding_size = 200\n",
    "step = 0\n",
    "learning_rate = 0.001\n",
    "\n",
    "#定义一个使用word-embedding查询同义词的函数\n",
    "#这个函数query_token是要查询的词，k表示要返回多少个最相似的词，embed是我们学习到的word-embedding参数\n",
    "#我们通过计算不同词之间的cosine距离，来衡量词和词的相似度\n",
    "def get_similar_tokens(query_token, k, embed):\n",
    "    W = embed.numpy()\n",
    "    x = W[word2id_dict[query_token]]\n",
    "    cos = np.dot(W, x) / np.sqrt(np.sum(W * W, axis=1) * np.sum(x * x) + 1e-9)\n",
    "    flat = cos.flatten()\n",
    "    indices = np.argpartition(flat, -k)[-k:]\n",
    "    indices = indices[np.argsort(-flat[indices])]\n",
    "    for i in indices:\n",
    "        print('for word %s, the similar word is %s' % (query_token, str(id2word_dict[i])))\n",
    "\n",
    "#将模型放到GPU上训练（fluid.CUDAPlace(0)），如果需要指定CPU，则需要改为fluid.CPUPlace()\n",
    "with fluid.dygraph.guard(fluid.CUDAPlace(0)):\n",
    "    #通过我们定义的SkipGram类，来构造一个Skip-gram模型网络\n",
    "    skip_gram_model = SkipGram(\"skip_gram_model\", vocab_size, embedding_size)\n",
    "    #构造训练这个网络的优化器\n",
    "    adam = fluid.optimizer.AdamOptimizer(learning_rate=learning_rate)\n",
    "\n",
    "    #使用build_batch函数，以mini-batch为单位，遍历训练数据，并训练网络\n",
    "    for center_words, target_words, label in build_batch(\n",
    "        dataset, batch_size, epoch_num):\n",
    "        #使用fluid.dygraph.to_variable函数，将一个numpy的tensor，转换为飞桨可计算的tensor\n",
    "        center_words_var = fluid.dygraph.to_variable(center_words)\n",
    "        target_words_var = fluid.dygraph.to_variable(target_words)\n",
    "        label_var = fluid.dygraph.to_variable(label)\n",
    "\n",
    "        #将转换后的tensor送入飞桨中，进行一次前向计算，并得到计算结果\n",
    "        pred, loss = skip_gram_model(\n",
    "            center_words_var, target_words_var, label_var)\n",
    "\n",
    "        #通过backward函数，让程序自动完成反向计算\n",
    "        loss.backward()\n",
    "        #通过minimize函数，让程序根据loss，完成一步对参数的优化更新\n",
    "        adam.minimize(loss)\n",
    "        #使用clear_gradients函数清空模型中的梯度，以便于下一个mini-batch进行更新\n",
    "        skip_gram_model.clear_gradients()\n",
    "\n",
    "        #每经过100个mini-batch，打印一次当前的loss，看看loss是否在稳定下降\n",
    "        step += 1\n",
    "        if step % 100 == 0:\n",
    "            print(\"step %d, loss %.3f\" % (step, loss.numpy()[0]))\n",
    "\n",
    "        #经过10000个mini-batch，打印一次模型对eval_words中的10个词计算的同义词\n",
    "        #这里我们使用词和词之间的向量点积作为衡量相似度的方法\n",
    "        #我们只打印了5个最相似的词\n",
    "        if step % 10000 == 0:\n",
    "            get_similar_tokens('one', 5, skip_gram_model.embedding._w)\n",
    "            get_similar_tokens('she', 5, skip_gram_model.embedding._w)\n",
    "            get_similar_tokens('chip', 5, skip_gram_model.embedding._w)\n",
    "            \n",
    "            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "从打印结果可以看到，经过一定步骤的训练，loss逐渐下降并趋于稳定。同时也可以发现skip-gram模型可以学习到一些有趣的语言现象，比如：跟who比较接近的词是\"who, he, she, him, himself\"。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Skip-gram的有趣使用\n",
    "在使用word2vec模型的过程中，研究人员发现了一些有趣的现象。比如当得到整个词表的word embedding之后，对任意词都可以基于向量乘法计算跟这个词最接近的词。我们会发现，word2vec模型可以自动学习出一些同义词关系，如：\n",
    "\n",
    "```shell\n",
    "Top 5 words closest to \"beijing\" are:\n",
    "1. newyork\n",
    "2. paris\n",
    "3. tokyo\n",
    "4. berlin\n",
    "5. soul\n",
    "\n",
    "...\n",
    "\n",
    "Top 5 words closest to \"apple\" are:\n",
    "1. banana\n",
    "2. pineapple\n",
    "3. huawei\n",
    "4. peach\n",
    "5. orange\n",
    "```\n",
    "\n",
    "除此以外，研究人员还发现可以使用加减法完成一些基于语言的逻辑推理，如：\n",
    "\n",
    "```\n",
    "Top 1 words closest to \"king - man + women\" are\n",
    "1. queen\n",
    "\n",
    "...\n",
    "\n",
    "Top 1 words closest to \"captial - china + american\" are\n",
    "1. newyork\n",
    "```\n",
    "还有更多有趣的例子，赶快使用飞桨尝试实现一下吧。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 思考一下\n",
    "\n",
    "[1] 如何使用飞桨实现CBOW算法。\n",
    "\n",
    "[2] 有些词天然具有歧义，比如“苹果”，在学习word2vec的时候，如何解决和区分歧义性词。\n",
    "\n",
    "[3] 如何构造一个自然语言句子的向量表示。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 引用\n",
    "\n",
    "[1] [Linguistic Regularities in Continuous Space Word Representations](https://www.aclweb.org/anthology/N13-1090/)\n",
    "\n",
    "[2] [维基百科：word2vec](https://en.wikipedia.org/wiki/Word2vec)\n",
    "\n",
    "[3] [text8数据集](http://mattmahoney.net/dc/textdata.html)\n",
    "\n",
    "[4] [知乎：Skip-gram和CBOW的优缺点](https://www.zhihu.com/question/68112508)\n",
    "\n",
    "## 课外阅读\n",
    "\n",
    "[1] [语言表示的前世今生](https://mp.weixin.qq.com/s/I315hYPrxV0YYryqsUysXw)\n",
    "[2] [词向量的本质](https://papers.nips.cc/paper/5477-neural-word-embedding-as-implicit-matrix-factorization.pdf)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PaddlePaddle 1.6.0 (Python 3.5)",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
